Imaging With Detection Routing

ABSTRACT

An imaging system includes a sensor array of sensor elements to convert incident light to detections. Plural detection accumulators are provided to store detections. Switches alternatively route detections from a sensor element selectively to n≧3 members of a set of said detection accumulators. An imaging process includes converting incident light to detections, and routing detections so as to partition them among n≧3 accumulators.

BACKGROUND

In a virtual-object collaborative environment, remote collaborators caninteract with and modify a shared virtual object. The shared virtualobject can be in the form of visible images, instances of which arepresented locally to respective collaborators. Interactions with thevirtual object can be effected using human input devices. For example,the virtual object can be a virtual document page; a collaborator canannotate the virtual document page using an IR pen (a stylus with a tipthat emits infra-red light). The annotations can then be presented toremote collaborators. In other words, the images of the document pagesat the different locations can be reconciled to the effect that they allrepresent the same object.

BRIEF DESCRIPTION OF THE DRAWINGS

The following figures represent examples and not the invention itself.

FIG. 1 is a schematic diagram of an imaging system in accordance with anexample.

FIG. 2 is a flow chart of a process in accordance with an example.

FIG. 3 is a schematic diagram of a collaborative environment inaccordance with an example.

FIG. 4 is a schematic diagram of an imaging system of the collaborativeenvironment of FIG. 3.

FIG. 5 is a flow chart of a collaborative process in accordance with anexample.

DETAILED DESCRIPTION

PCT Patent Application number PCT/US11/58896 filed 2011 Nov. 2 andentitled “Projection Capture System, Programming And Method” discloses avirtual-object collaborative environment (e.g., a mixed reality system)with respective imaging systems provided for each collaborator. Eachimaging system includes a projector for projecting a visible imagerepresenting a virtual object, e.g., a document page. Each imagingsystem includes a visible-light camera to track changes in the virtualobject, e.g., due to actions by the local collaborator using an IR pen.To this end, an IR camera is used to track the position and motion ofthe IR pen.

Such a system must address the challenge of color registration. First ofall, the images taken by the IR camera must be aligned (registered) withthe visible light image. Furthermore, if the visible-light camera usesseparate sensor elements for color components (e.g., red, green, andblue) of the visible light, the resulting monochrome images must beregistered to avoid undesirable artifacts in the composite image.Calibration and post-processing procedures may be available to addressregistration problems, but they can incur undesirable penalties in timeand processing power. For example, the post-processing can increase thelatency required to update a projected visible image based on a captureddigital image.

Examples disclosed herein address registration challenges in part byusing full-range sensor elements to capture all colors, albeit atdifferent times. As explained below with respect to imaging system 100(FIG. 1), detections output by each sensor element are routed to atleast three different color-dedicated accumulators as a function of timeso that each accumulator accumulates detections for a respective color.As explained further below with reference to collaborative environment300 (FIGS. 3 and 4), the inclusion of an accumulator dedicated to IRdetections addresses the problem of registration of the IR light to thevisible light and provides for interleaving of the accumulations tominimize time-based registration problems even in the case of sceneswith moving elements, e.g., a moving IR pen.

An imaging system 100, shown in FIG. 1, includes a sensor array 102 ofsensor elements 104. Each sensor element 104 is to convert incidentlight 106 to detections 108. Depending on the type of technologyemployed, detections 108 may take the form of electric charges or someother form.

In addition to sensor elements 104, imaging system 100 includesdetection accumulators 110. Associated with each sensor element 104 is aset of n detection accumulators, where n is an integer greater than orequal to three (n≧3) so that, for example, at least a first detectionaccumulator 110R can be dedicated to accumulating detections of redlight, at least a second detection accumulator 110G can be dedicated toaccumulating detections of green light, and at least a third detectionaccumulator 110B can be dedicated to detections of blue light.Alternatively, each of three or more detection accumulators can bededicated to a respective dimension of a color space other thanred-green-blue (RGB). Imaging system 100 further includes a switch set112 of switches 114. Each switch 114 is to route detections from asensor element 104 to detection accumulators 110 as a function of thecolor of the incident light being converted to the detections.

An imaging process 200 that can be implemented using imaging system 100or another imaging system is flow charted in FIG. 2. At 201, incidentlight (e.g., incident a sensor element 104) is converted to detections.At 202, the detections are routed (e.g., by switch set 114) to“color-dedicated” detection accumulators 100 as a function of the colorof the incident light being converted to the detections.

The use of plural color-dedicated accumulators advantageously decouplesthe process of inputting detection to the accumulators from the processof outputting accumulations from accumulators. If only one accumulatoris used for plural colors, the accumulator must be read out and resetwhenever the color to be detected is changed. When pluraldedicated-color accumulators are used, one color can be read whileanother is accumulated. Also, when plural accumulators are used (persensor element), accumulations can be “interleaved”, i.e., a first colorcan be accumulated, then another color can be accumulated, and then thefirst color can be accumulated again, all without an intervening readoutor reset.

For example, to achieve a frame duration of 30 milliseconds (ms) using asingle accumulator, the following pattern could be used: 10 ms of red,readout and reset, 10 ms of green, readout and reset, 10 ms of blue,readout and reset, and repeat. If there are moving elements in the sceneor object being imaged, there will be an average of about 15 ms ofopportunity for color misregistration due to motion in the scene beingimaged. The examples described herein reduce this opportunity for colormisregistration due to scene motion.

Plural color-dedicated accumulators would permit an interleaving patternsuch as 5 ms red, 5 ms green, 5 ms blue, 5 ms red, 5 ms green, 5 msblue, readout and reset, and repeat. In this case, the average timeavailable for misregistration is reduced by half to about 7.5 ms. Thetime available for misregistration due to scene movement can be reducedto less than 2 ms by cycling through 1 ms phases for each color. In thatcase, the average misregistration between colors is limited by theamount of scene movement that occurs with 2 ms. Thus, the interleavingsprovided for by using plural dedicated color accumulators can readilyreduce color misregistration by an order of magnitude or more. Furtheradvantages are attainable when infra-red capabilities are integratedinto the camera, as in the following example.

A collaborative environment 300 is illustrated schematically in FIG. 3including imaging systems 310 and 320 coupled by a network 330. Whiletwo imaging systems are represented in FIG. 3, other examples includeother numbers of imaging systems. Imaging system 310 includes an RGBprojector 312, an RGB-IR camera 314, and an IR pen 316. Imaging system320 is essentially similar to imaging system 310; imaging system 320includes an RGB projector 322, an RGB-IR camera 324, and an IR pen 326.

Imaging system 310 can be used to generate a machine-readable digitalimage 332 of a physical object 334 (which can be, for example, athree-dimensional object or a page of a document). To this end, RGBprojector 312 can be used to illuminate physical object 334, and RGB-IRcamera 314 can capture light reflected by physical object 334 to producedigital image 332. Digital image 332 can then be input to RGB projector312 so that RGB projector 312 projects (e.g., generates) ahuman-perceptible visible image 336.

Visible image 336 can serve as a virtual object that can bemanipulated/modified by a user, e.g., using IR pen 316 as well as otherhuman-input devices. IR pen 316 is a stylus with a tip that emits IRlight; the tip may also emit visible light to let a user known when theIR light is active. A local collaborator (or other user) can use IR pen316, for example, to control the position of a cursor in visible image336 and to manipulate or annotate visible image 336 by gesturing or“writing” with IR pen 316. RGB-IR camera 314 detects the position of IRpen 316 so that the position of IR pen 316 can be tracked. Commands canbe implemented and cursor position adjusted in digital image 332; as itis updated, digital image 332 can be input to RGB projector 312 toupdate visible image 336.

In addition to its use locally with respect to imaging system 310,digital image 332 can be communicated over network 330 to imaging system320. There, digital image 332 can be input to RGB projector 322 togenerate visible image 338. Like visible image 336, visible image 338can be manipulated by a remote collaborator (or other user). e.g., usingIR pen 326. RGB-IR camera 324 can track the position of IR pen 326 andupdate digital image 332. In practice, each imaging system 310, 320maintains an instance of digital image 332; programs on the imagingsystems ensure that both instances are updated so that visible images336 and 338 are reconciled (i.e., synchronized) in near real time. Thus,local and remote collaborators can work together to dialog about andedit the virtual object represented by visible images 336 and 338.

Collaborative environment 300 and its imaging systems 310 and 320 havesome applications and some elements in common with their counterpartsdisclosed in PCT Patent Application number PCT/US11/58896 entitled“Projection Capture System, Programming And Method”. However, imagingsystems disclosed in that application used separate cameras for visiblecolors and for IR. RGB-IR cameras 314 and 324 combine visible colorcapture and IR capture for simplicity, economy, and better registrationof color and IR image components.

RGB-IR camera 314, as shown in FIG. 4, includes a channel array 400 ofdetection channels 402. Each detection channel 402 includes a respectivesensor element 404, a respective switch 406, and a respectiveaccumulator set 408. Accordingly, channel array 400 includes atwo-dimensional sensor array 410 of sensor elements 404, atwo-dimensional switch array 412 of switches 406, and a two-dimensionalaccumulator-set array 414 of accumulator sets 408. Sensor array may beformed on the back side of a backside-illuminated CMOS sensor, whileswitch array 412 and accumulator-set array 414 may be formed on thefront side of the backside-illuminated CMOS sensor. Each accumulator set408 includes plural color-dedicated accumulators 416, in this case, ared-dedicated accumulator 416R, a green-dedicated accumulator 416G, ablue-dedicated accumulator 416B, and an IR-dedicated accumulator 416J.Accumulators 416 can be implemented as integrating capacitors.

Sensor elements 404 are “full-range” in that each sensor element 404 candetect incident red, green, blue, and infra-red light. In response todetection of incident light, e.g., in the form of photons, each sensorelement 404 outputs “detections”, in this case, in the form ofelectrical charges. All accumulators 416 are essentially similar; thediffering labels reflect the fact that, in use, each accumulator isdedicated to a respective color (e.g., red. green, blue, and infra-red).

Each switch 406 has outputs coupled to inputs of respective accumulators416 of a respective accumulator set 408. Accumulator outputs are coupledto readout circuitry 418, which includes an analog-to-digital converter(ADC) for converting analog accumulator values to digital values. Inother examples, ADCs are located between accumulators and readoutcircuitry.

The output of readout circuitry 418 is received by an image-data handler420, e.g., a computer, which can use the received digital data toconstruct and update digital image 332. Digital image 332 iscommunicated from image-data handler 420 to RGB projector 312 to updatevisible image 336. In addition, image-data handler 420 communicates vianetwork 330 with its counterparts in other imaging systems, e.g.,imaging system 320, to reconcile (i.e., resolve differences between,synchronize, equalize) instances of digital image 332.

A timing controller 422 controls and synchronizes (i.e., coordinates thetimings for) switches 406, readout circuitry 418, and red, green, andblue emitters 424R, 424G, 424B of RGB projector 312. Each switch 406 issynchronized with projector 312 so that, at any given time, detectionsare routed to the accumulator corresponding to the color being emittedby projector 312. During gaps in emissions by projector 312, IRdetections are routed to IR accumulators 416J. Thus detections arerouted to accumulators as a function of the color of the incident lightfrom which the detections resulted. Timing controller 422 can includecircuitry within RGB projector 312, within RGB-IR camera 314, and/orexternal to both RGB projector 312 and RGB-IR camera.

An imaging process 500, flow charted in FIG. 5, can be implemented usingcollaborative environment 300 or another environment. At 501, timings ofan imaging system are controlled so that emitting by a projector,switching within a camera, and readout from the camera are synchronized(i.e., their timings are coordinated). In the context of collaborativeenvironment 300, timing controller 422 is configured to synchronizetimings used for operation of projector 312, camera switches 408, andreadout circuitry 418. For example, while timing controller 422 iscausing RGB projector 312 is emitting red, switch 406 is directing(presumably red) detections to red accumulator 416R. For anotherexample, while timing controller 422 is causing RGB projector 312 to notemit any color, timing controller 422 causes switch 406 to direct(presumably IR) detections to IR accumulator 416J. Also, timingcontroller 422 causes readout circuitry 418 to read out respectivelyfrom accumulators 416R, 416G, 416B, and 416J, only while they are notreceiving detections.

For example, RGB projector 312 can emit colors sequentially to yieldemitted visible light 428, resulting in visible light 430 to be incidentto camera 314 due to reflections of emitted visible light 428. Due tocontrol of RGB projector 312 by timing controller 412, incident visiblelight 430, like emitted visible light 428, can consist of cyclicalphases, e.g., a red phase 430R, a green phase 430G, a blue phase 430B,and a gap phase 430X. Switches 406 route detections during red phases430R to red accumulator 416R, detections during green phases 430G togreen accumulator 416G, detections during blue phases 430B to blueaccumulator 416B, and detections during gap phases 430X to IRaccumulator 416J. The phases can have different durations andfrequencies. For example, a pattern RGBXRGBXRGBX, etc., allows theposition of IR PEN 316 (which may be moving) can be sample morefrequently than visible image 336 (which may be a stationary document).

At 502, colors are emitted sequentially, i.e., one at a time. Forexample, the sequence can be RGBRGBRGB . . . in which a sequence RGB isrepeated; alternatively, a sequence may not have a repeating pattern. Ina case such as imaging system 310 (in which RGB-IR camera 314 detectsmore colors than RGB projector 312 emits), the pattern can include gaps,e.g., RGBXRGBX or RXGXBXRXGXBX, e.g., to allow a camera to detect lightfrom a different source (such as infra-red from an IR pen). The number mof different colors represented is greater than or equal to three sothat a full color image can be obtained by integrating over thedifferent colors. The light emitted over each color phase of each cyclecan be uniform (e.g., to illuminate a physical object) or image-bearing(e.g., when projecting an image).

At 503, a collaborator or other entity interacts with a scene using ahuman interface device such as IR pen 316. The scene can include aphysical object, e.g., illuminated by RGB projector 312, and/or avisible image, e.g., projected by projector 312. For example, acollaborator may use IR pen 316 to point to a portion of visible image336; collaborative environment 300 can then reconcile visible image 338so that collaborators using imaging systems 310 and 320, respectively,can focus on the same area of the common virtual object. Also, IR pen316 can be used to annotate a document image (or even a physical object)or issue commands (e.g., “rotate”, “zoom”, etc.).

At 504, incident light (visible and IR) is detected by sensor array 410,resulting in detections. At 505, switches are operated to directdetections to n≧m color-dedicated accumulations. Switches 406 can becontrolled by timing controller 422 in synchronization with RGBprojector 312 so that the detection are routed to the accumulator 416R,416G, 416B, or 416J corresponding to the phase of the incident lightcausing the detections. For example, while RGB projector 312 is emittingred light, switches 406 are set so that detections are routed to “red”accumulator 416R. For another example, during gaps between coloremissions by RGB projector 312, switches 406 are set so that detectionsare routed to IR accumulator 416J. Note that the color phases can varyin number and/or duration by color. For example, there can be two greencycles or a double-length green cycle to take advantage of the fact thatgreen is perceived as most closely related to intensity (brightness) towhich human eyes or more sensitive than hue.

At 506, accumulations are interleaved. Herein, accumulations are“interleaved” when an accumulation within one accumulator coupled to aswitch to receive detections from a sensor includes detections acquiredboth before and after detections accumulated within another accumulatorcoupled to the switch to receive detections from the sensor.“Interleaving” can include single-color interleaving, e.g., RGBG, inwhich only one accumulator (in this case, green accumulator (416G) isinterleaved, and all-color interleaving, e.g., RGBRGB in which allaccumulations are interleaved. Interleaving is made possible by thepresence of dedicated-color accumulators (as opposed to using a singleaccumulator that must be emptied before it can be used to storedetections associated with the next color). As explained below,interleaving makes possible dramatic reductions in problems with colormisregistration due to moving elements in a scene.

At 507, accumulations are read out; once its contents have been read, anaccumulator is reset (e.g., to zero). For embodiments that do not employinterleaving, accumulators can be read out and reset each color cycle,e.g., RGB, or RGBJ. For embodiments that do employ interleaving,readouts/resets can occur after plural color cycles, e.g., RGBRGB orRGBJRGBJ (where J can correspond to switch settings in which projector312 is not emitting a color and detections routed are to IR accumulator416J.

In some examples, the number of color-phase cycles (e.g., RGBXinstances) can be large, e.g., tens or hundreds between readouts so thatthe accumulations in a set of accumulators are highly overlapped in timeso as to minimize misregistration of colors in a digital image (withoutrequiring color registration post processing, which can be timeconsuming and, thus, delay collaborative image updating). In the courseof 507, readout circuitry 418 can convert analog values stored in theaccumulators to digital values for use by image-data handler 420.

At 508, a digital image 332 is created/updated based on the data readout from the accumulators. At 509 the image data is interpreted, e.g.,by image data handler 420 to track IR pen position and motion. Notethat, if the IR pen is to be tracked against a stationary object orprojected image, the pen position can be sampled more frequently thanthe object or projected image. For example, projector 312 can emitrepetitions of RXGXBX and detections can be routed with repetitions ofthe pattern RJGJBJ.

At 510, instances of a digital image at different imaging systems arereconciled. For example, copies of digital image 332 (FIG. 3) stored byrespective imaging systems 310 and 320 can be reconciled. At 511, thereconciled digital images can be input to respective projectors togenerate/update visible images. Since digital images are reconciled, thevisible images generated from them are also reconciled. For example,visible images 336 and 338 (FIG. 3) are reconciled so that collaboratorsusing respective imaging systems 310 and 320 can operate on the samevirtual objects.

In other examples, other color sets are used. For example, ultra-violetlight may be detected instead of or in addition to infra-red. Detectioncolor phases may be longer and/or more frequent than others. In someexamples, there are more than one accumulator per color per sensor. Forexample, plural IR accumulators can be used to detect phase and thusdepth of an IR pen.

In the illustrated examples, m<n, i.e., the number m of emitters in theprojector involved in time-sequential emissions is less than or equal tothe number n of accumulators per accumulator set, i.e., per sensorchannel. For example, three accumulators can be used with a projectorthat emits three colors (RGB), or four accumulators can be used with aprojector that emits four colors (RGB-IR), or four accumulators can beused with a projector that emits three colors (RGB), with the fourthaccumulator used to detect IR from a source other than the projector.

In alternative examples, m>n. For example, a projector can have m>nemitters, but limit the number used to n. Thus, a projector may haveemitters for R, G, B, IR, and UV, but use only one of IR and UV at atime. For another example, a projector can have six emitters: red,green, blue, magenta, cyan, and yellow, and three accumulators can beused during even cycles for red, green, and blue and during odd cyclesfor magenta, cyan, and yellow. In the illustrated examples, eachaccumulator is dedicated to accumulating detections only for a singlecolor (R, G, B, or IR). However, in some examples, e.g., where m>n, thecolor accumulator by an accumulator may be changed, e.g., for differentreadout cycles.

Herein, a “system” is a set of interacting non-transitory tangibleelements, wherein the elements can be, by way of example and not oflimitation, mechanical components, electrical elements, atoms, physicalencodings of instructions, and process segments. Herein “device” refersto a hardware or hardware+software system. Herein, “process” refers to asequence of actions resulting in or involving a physical transformation.An “imaging process” is a process for creating visible and/or digitalimages.

An “imaging system” is a system that creates visible and/or digitalimages. Herein, “image” refers to a (uniform or non-uniform) spatialdistribution of light or a digital representation of such a distributionof light. A “visible image” is an image that a human can perceive; a“digital image” is a non-transitory tangible encoding of data in digitalformat that represents a visible image (and may include other data).

Herein, a “virtual object” is a digitally-defined object that representsa human-manipulable object and that can be manipulated by a human as ifit were that manipulable object. For example, a projected image of adocument can represent a hardcopy document and can be manipulated, e.g.,annotated using an IR pen, (more or less) as if it were the hardcopydocument.

Herein “light” is electromagnetic radiation. “Light” encompasses“visible light”, which consists of light within a wavelength rangeperceptible to the human eye, and “invisible light”, which is lightoutside the wavelength range perceptible to the human eye andencompasses “infra-red light” and “ultra-violet light”. A “sensor” is ahardware device for converting incident light (i.e., light that reachesthe sensor) into detections. A “sensor array” is a sensor constituted byan array, typically two-dimensional) of “sensor elements”, each of whichis a sensor in its own right.

Herein, a “detection” is a tangible entity produced in response toincident light. A detection can be, for example, an electrical charge oran set of electrical charges. Alternatively, a detection can be avoltage level or a light intensity level (where the sensor effectivelygenerates amplified light in response to incident light), or a detectioncan take another form.

Herein, a “detection accumulator” or just “accumulator” is a device thataccumulates or counts detections. For example, an accumulator can be anintegrating capacitor that increases its charge level as detections inthe form of electrical charges are received. In other examples, anaccumulator can take another form such as a counter that counts lightpulses generated in response to incident light.

Herein, “switch” refers to a device with an input, plural outputs, and acontrol port for receiving a signal that selects one of the outputs tobe connected to the input. In the present case, a switch input isconnected to an output of a sensor element for receiving a detectiontherefrom; each switch output is coupled to a respective color-dedicatedaccumulator to, when coupled to the switch input, direct the detectionto the respective accumulator; and each control port is coupled to thetiming controller so that the switch settings can be synchronized withprojector emitters. Herein, “partitioned” means “allocated”, in thesense that each detection is allocated to a respective accumulatoraccording to the switch setting at the time the detection is made.Herein, “set” requires at least two elements.

Herein, a “readout system” and “readout subsystem” refer to systems forreading out values from other devices, such as accumulators, e.g., todetermine the number or amount of detections accumulated by anaccumulator. “Reset” herein refers to initializing a device, e.g.,setting an accumulator so that the amount of detections it represents iszero.

Herein, “computer” refers to a hardware machine for processingphysically encoded data in accordance with physically encodedinstructions. A “server” is a computer that performs services for othercomputers. Depending on context, reference to a computer or server mayor may not include software installed on the computer. Herein, “storagemedium” and “storage media” refer to a system including non-transitorytangible material in or on which information is or can be encoded withinformation including data and instructions. “Computer-readable” refersto storage media in which information is encoded in computer-readableform.

In this specification, related art is discussed for expository purposes.Related art labeled “prior art”, if any, is admitted prior art. Relatedart not labeled “prior art” is not admitted prior art. In the claims,“said” introduces elements for which there is explicit verbatimantecedent basis; “the” introduces elements for which the antecedentbasis may be implicit. The illustrated and other described embodiments,as well as modifications thereto and variations thereupon are within thescope of the following claims.

What is claimed is:
 1. An imaging system comprising: a sensor array ofsensor elements to convert incident light to detections; detectionaccumulators; and switches to couple said detection accumulators to saidsensor elements, each switch selectively routing detections from asensor element so that they are partitioned among n≧3 members of a setof said detection accumulators to yield n concurrent accumulations. 2.An imaging system as recited in claim 1 further comprising a readoutsubsystem to read out said accumulations after they become interleaved.3. An imaging system as recited in claim 1 further comprising: anemitter subsystem to emit m colors sequentially; and a timing controllerto synchronize said switches with said emitter subsystem so that each ofsaid charge accumulators accumulates detections corresponding to at mostone of said colors, and so that different ones of said chargeaccumulators of said set accumulate detections for respective differentones of said colors.
 4. An imaging system as recited in claim 3 whereinn≧m.
 5. An imaging system as recited in claim 4 further comprising areadout subsystem to read out said accumulations after they becomeinterleaved.
 6. An image system as recited in claim 4 wherein n>m.
 7. Animaging system as recited in claim 6 wherein said m colors include red,green, and blue, and said n charge accumulators are used for separatelystoring red, green, blue, and infra-red or ultra-violet detections. 8.An imaging process comprising: detecting incident light using a sensorarray of sensor elements to yield detections; and operating switches toroute said detections to detection accumulators so that detections madeby a sensor element are partitioned for storage among a set of n≧3detection accumulators to yield n concurrent accumulations.
 9. Animaging process as recited in claim 8 further comprising reading outsaid n concurrent accumulations after they become interleaved.
 10. Animaging process as recited in claim 8 further comprising: sequentiallyemitting light of m colors; and synchronizing said switches with saidemitting so that detections of a portion of said light falling on saidsensor element is partitioned among the detection accumulators of saidset based on color of incident light from which respective detectionsare generated.
 11. An imaging process as recited in claim 10 whereinn≧m.
 12. An imaging process as recited in claim 11 further comprisingreading out said n concurrent accumulations after they becomeinterleaved.
 13. An imaging process as recited in claim 11 wherein n=m.14. An imaging process as recited in claim 13 wherein n>m.
 15. Animaging process as recited in claim 14 wherein said m colors includered, green, and blue, and said n charge accumulators are used forseparately storing red, green, blue, and infra-red or ultra-violetdetections.